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1.  OVERVIEW  AND  REPORT  ORGANIZATION 

This  report  provides  a  proposed  architecture  for  an  adaptive  Operator-Machine  Interface  (0^ 
appuZo  *e  specific  objectives  of  *is  Phase  I  SBIR  effort.  The  featores  of  an  adapnve  OMI 
a^d  other  considerations  that  must  be  addressed  are  also  included.  We  present  a  methodology 
for  using  advanced  cognitive  modeling  techniques  to  identify  the  knowledge  and  skill  requir  - 
ments  for  the  SH-60R  Sensor  Operator  (SENSO).  We  describe  the 

ina  the  SENSO  with  the  increased  sensors  and  capabilities  of  the  SH-60R  Multi-Mission  Heli¬ 
copter  Upgrade  (MMHU).  An  adaptive  OMI  is  proposed  as  a  method  to  alleviate  these  pro  - 

lems. 

We  propose  an  Integrated  Intelligent  Tutoring  Environment  (IITE)  as  an  environment  ^at  allows 
for  the  development  of  an  adaptive  OMI  as  well  as  an  Intelligent  Tutoring  System.  This 
ronment  will  use  a  Cognitive  Task  Analysis  (CTA)  to  identify  the  expertise  required  of  the 
SENSO  on  this  newly  evolving  platform.  Although  the  military  has  used- CTA  techmques  to 
identify  the  skill  and  knowledge  of  Subject  Matter  Experts  (SMEs),  these  techniques  can  be  very 
costly.  We  provide  a  description  of  an  automated  tool  that  can  be  used  to  conduct  a  CTA  and 
detail  how  this  tool  can  be  used  to  capture  the  SENSO’s  expertise. 

Implementation  of  an  adaptive  OMI  also  requires  a  means  for  capturing  each  user’s  current 
knLedge  state.  A  component  of  DSR’s  Integrated  Training  System  (ITS)  is  presented  as  an 
option  to  achieve  this  goal.  The  ITS,  developed  under  SBIR  N91-052,  has  a  diagnostic  compo¬ 
nent  that  describes  the  user’s  skill  level  as  compared  to  the  expert  model.  Specific  techniques  for 
adapting  the  OMI  are  addressed  including  embedded  coaching,  electronic  references,  he^p  tools, 
and  customized  tools  such  as  automation  techniques.  The  expert  model  developed  for  the  adap¬ 
tive  OMI  can  also  be  used  as  the  backbone  for  an  intelligent  tutoring  system  and  an  expert  sys¬ 
tem  to  support  decision-making  tasks.  We  further  describe  the  hardware  and  software  required  to 
implement  such  a  system.  The  implementation  and  training  of  an  adaptive  OMI  present  a  unique 
challenge.  Our  recommendations  for  achieving  these  tasks  are  also  presented. 

A  summarN^  of  the  Phase  I  SBIR  is  presented  in  Section  2.  We  found  that  an  OMI  that  adapts  to 
the  current  user  has  great  potential  and  application  for  the  SH-60R  sensor  operator  station,  ur 
recommendations  for  Phase  II  research  are  presented  in  Section  3.  We  propose  to  develop  a 
prototype  adaptive  interface  for  the  SENSO.  By  conducting  a  cognitive  task  analysis  on  a^x- 
isting  sensor  subsystem,  the  expert’s  mental  model  of  that  subsystem  will  be  specified.  That 
model  can  then  be  used  to  adapt  the  subsystem’s  interface  to  the  user’s  skill  level. 

Section  4  describes  the  components  of  an  adaptive  OMI.  Considerable  detail  is  provided  on  the 
cognitive  task  analysis  process  and  the  Cognitive  Analysis  Tool  (CAT)  that  can  be  used  to  con¬ 
duct  the  CTA.  A  thorough  description  of  the  potential  design  opUons  for  an  adaptive  OMI  are 
also  presented  in  this  section.  These  include  techniques  that  can  be  used  when  the  system  infers 
that  the  current  user  has  acted  erroneously. 
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Section  5  describes  the  conceptual  architecture  of  the  proposed  system.  Devel^ment  of  the 
adaptive  OMI  software  is  best  completed  concurrently  with  the  tacucal  software,  ms  is  a  Ph^e 
II  issue  and  one  which  includes  careful  analysis  and  consideration  of  software  and  hardware  de¬ 
velopment  and  hosting  techniques  which  maximize  the  use  of  COTS  and  software  fansportabil- 
itv  such  as  Middleware  used  by  DSR  in  the  Multi-Purpose  Processor  (MPP)  acoustic  processing 
program.  The  implementation  of  an  adaptive  OMI  is  described  in  Section  6.  Sigmficant  detail  is 
provided  on  the  most  effective  way  to  train  the  users  to  efficiently  use  an  adaptive  system.  In 
Lction  1,  future  directions  as  applied  to  the  SENSO’s  adaptive  OMI  are  discussed. 


1.1  Introduction 

The  last  two  decades  have  produced  major  increases  in  the  sensitivity,  reliability,  efficiency,  and 
complexity  of  sensors  available  to  Navy  operators  and  tacticians.  While  those  increased  capa¬ 
bilities  have  provided  distinct  attendant  warfighting  advantages,  the  sensor  processing  and  man¬ 
agement  requirements  now  needed  to  counter  today’s  threats  pose  a  major  challenge  to  the  air 
avionics  sensor  operator  due  to  both  the  complexity  of  the  sensors/weapons  systems  and  a 
greatly  increased  task  loading. 

Airborne  avionics  sensor  operator  tasks  are  complex  and  severely  loaded  by  time  limitations  and 
high  information  flow  and  presentation.  The  recent  emphasis  on  Littoral  Warfare,  with  even 
higher  contact  information  rates  and  rapidly  changing  threats,  requires  yet  even  faster  operator 
decisions  and  improved  operator-system  performance.  Examples  of  the  different  mis^ons  that 
the  SENSO  may  engage  in  include  Anti-Submarine  Warfare,  Anti-Surface  Warfare,  Over-the- 
Horizon  Targeting,  and  Search  and  Rescue.  As  an  example  of  the  complexity  involved,  these 
tasks  can  be  characterized  as  typical  of  an  ASW  acoustic  sensor  operator  simultaneously  moni¬ 
toring  all  or  some  of  the  narrow  band,  broad  band,  transient  and  active  receive  buoy  processing 
modes  for  from  between  8  to  32  independent  buoys  each  with  1  to  4  beams  of  data.  In  addition, 
each  of  these  modes  has  many  processing  parameter  options  -  such  as  firequency  resolutions,  in¬ 
tegration  times,  bandwidth  searched,  time  resolutions  and  active  signal  designs  -  that  must  be 
selected  based  on  the  environmental  and  tactical  scenarios.  Uncertainty  as  to  which  parameter 
value  should  be  chosen  often  leads  to  parallel  processing  of  multiple  parameter  values,  which  in 
turn  increases  the  data  rate  of  the  information  that  the  operator  must  monitor.  Paging  through 
gram  data  for  detection  purposes  with  multiple  narrow  band  frequency  resolutions  and  broad 
band  delay  grams  for  multiple  beam  buoy  data  can,  just  by  itself,  all  too  easily  overload  sensor 
operators.  Additional  operator  tasks  including  system  processing  mode  setup  (processing  mode 
and  parameter  selections),  classification,  tracking  and  localization  will  force  operators  to  com¬ 
promise  performance  on  some  tasks  in  order  to  complete  the  required  string  of  tasks. 

This  SBIR  seeks  to  reduce  this  problem  of  operator  workload/efficiency  by  increasing  operator 
skill  levels  through  continually  tracking  the  operator’s  performance,  comparing  that  performance 
to  an  ideal  performance  model  and  using  that  comparison  to  provide  the  operator  real-time  and 

after-the-event  assistance. 
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Today’s  SENSO  has  limited  oppornmity  for  realistic  Anti-Submarine  Warfare  (ASW)  practice 
and  experience  through  actual  “cold  war"  encounters.  Training  for  new  operators  provides  the 
SENSO  with  basic  system  operation  capability.  Moreover,  in  environments  where  ex^nenced 
mentors  (watch  supervisors,  etc.)  are  available,  these  new  operators  are  assisted  while  they  gam 
the  experience  needed  to  perform  the  complex  tasks  requited  to  quickly  and  accurately  detect 
and  classify  contacts.  The  SENSO  operating  environment  m  the  helicopter  requires  addition^ 
independence  and  more  rapid  decision-making.  This  motivates  a  need  for  a  flexible  SENSO 
OMl  matched  to  individual  operator  capabilities  for  multiple  sensor  systems.  A  flextble  OMl 
can  assist  the  SENSO  in  making  rapid,  appropriate,  real  time  decisions  for  detection,  contact 
evaluation,  tracking,  localization,  and  operational  mode.  WhUe  adaptive  “ve  ^ 
research  subject  for  several  years,  they  have  not  been  wtdely  applied.  Recent  studies  h 
shown  that  participants  tend  to  have  much  greater  task  performance  and  task  satisfaction  when 

using  adaptive  interfaces  (Gong  &  Salvendy,  1995). 

This  Phase  1  report  describes  a  proposed  adaptive  OMl  technique  to  enhance  the  P«fo™ance  of 
the  inexperienced  SENSO  by  adapting  the  OMl  to  match  the  operator’s  skill  level.  This  will  be 
accomplished  using  a  cognitive  task  analysis  approach  derived 

experts  in  the  Cognitive  Science  field  (Kieras  &  Poison,  1985;  Anderson,  1983,  Willtams  & 
Reynolds  1990)  The  essential  information  required  for  a  cognitive  task  analysis  is  obtained 
from  experienced,  highly  skilled  SENSOs.  This  skill  information  establishes  the  ideal  or  expert 
model  against  which  each  individual  operator  is  compared  -  providing  a  dia^osis  of  each 
SENSO’s  skill  level.  The  individual  operator’s  skill  level  will  feed  an  Adaptive  OMl  Processor 
that  enhances  the  system’s  interface.  The  goal  of  this  adaptive  interface  is  to  improve  the  overall 
performance  envelope  by  providing  optimal  mode  configuration  and  automated,  on-screen  op¬ 
erator  aids. 
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2.  PHASE  I  SUMMARY 


2.1  Objectives 

DSR’s  Phase  I  SBIR  objective  was  to  provide  a  proposed  baseline  architectoe  that  maximizes 
system  performance  through  an  operator-machine  interface  that  adapts  to  different  operator  sb 
levels  and  varying  degrees  of  system  automation.  Recent  technological  advances  have  resulted 
in  increases  ^system  capabilities  and  system  complexity.  This  increased  complexity,  however 
has  not  necessarily  been  accompanied  by  increased  support  for  the  system 
systems  in  particular  have  significantly  increased  in  complexity.  One  way  to  assist  the  SENSO  m 
efficiently  utilizing  these  systems  is  to  provide  a  Hexible  user  interface.  A  flexible  interface  is 
one  that  adapts  the  presentation  of  information  and/or  the  methodology  for  obtaining  inputs  from 
the  user  based  on  the  specific  tasks  to  be  accomplished  or  the  characteristics  or  preferences  of  the 

user. 

There  are  two  types  of  flexible  interfaces:  adaptable  and  adaptive.  An  adaptable  system  provides 
the  user  with  the  tools  to  change  the  system  setup.  Many  of  today’s  programs  wntten  for  e 
Microsoft  Windows  Operating  System  would  be  described  as  adaptable  since  the  user  has  the 
option  to  display  the  buttons  that  activate  the  preferred  functionality.  An  adaptive  interface,  on 
the  other  hand,  changes  its  own  characteristics  automatically  depending  upon  the  needs  of  the 
user.  Evidence  indicates  that  users  either  have  difficulty  using,  or  choose  not  to  use,  adapteble 
features  (Oppermann,  1994).  Self-adaptive  systems,  on  the  other  hand,  bring  inherent  problems 
such  as  determination  of  the  best  way  to  adapt  the  system  given  the  current  circumstances. 


2.2  Phase  I  Summary 

The  objective  of  this  Phase  I  effort  was  to  describe  a  methodology  for  developing  an  adaptive 
OMI  for  the  Sensor  Operator  station  on  the  SH-60R  MMHU.  At  the  kick  off  meeting  for  this 
effort  the  government  informed  the  Phase  I  SBIR  contractors  that  considerable  unce^nty  ex- 
isted  in  the  SH-60R  system  definition.  It  was  recommended  that  the  effort  under  this  Phase  I  fo¬ 
cus  on  the  characteristics  of  the  SENSO  operators  and  deal  more  generically  with  the  syaem  in¬ 
terface  which  would  be  more  in  focus  for  the  Phase  II  effort.  This  was  accomplished  by  re- 
viewing  existing  research  in  several  areas  including; 

•  Cognitive  Task  Analysis  techniques 

•  Adapjive  User  Interfaces 

•  Intelligent  User  Interfaces 

•  Performance  Support  Systems 

•  Expert  Systems 

•  Intelligent  Tutoring  Systems 
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The  research  revealed  that  although  adaptive  interfaces  have  been  the  focus  of  research  for 
years,  they  have  not  been  widely  implemented.  The  main  reason  for  this  is  the  difficulty  in¬ 
volved  in  conducting  the  cognitive  task  analysis.  Moreover,  the  integration  of  such  a  system  in¬ 
volves  complicated  training  and  implementation  issues. 

There  are  several  goals  that  can  be  achieved  with  the  implementation  of  an  adaptive  interface, 
including: 

•  Enhanced  user  productivity  in  that  users  are  provided  with  the  optimal  level  of  assis¬ 
tance  in  their  transition  from  novice  to  expert. 

.  Increased  suitability  of  the  system  for  specific  tasks  in  that  users  could  more  easily 
access  functions  and  perform  tasks  they  may  be  weak  in. 


•  Optimized  workload. 

•  Increased  user  satisfaction  in  that  users  may  have  access  to  different  functions  with¬ 
out  extensive  training. 

Each  of  these  goals  is  certainly  relevant  and  desirable  in  most  situations  involving  complex  tacti¬ 
cal  equipment.  The  increased  complexity  of  the  SENSO’s  watchstation  has  the  potential  to  sig¬ 
nificantly  impact  the  user’s  productivity.  The  increased  workload  that  will  accompany  the  mul¬ 
tiple  and  varied  sensor  systems  that  are  to  be  integrated  on  this  new  helicopter  vanant  must  be 
handled  proactively.  An  adaptive  interface  is  one  method  that  can  help  achieve  this. 

The  proposed  structure  of  this  adaptive  OMI  utilizes  an  automated  tool  to  perform  a  CTA  The 
CTA  provides  the  content  of  an  expert  model  of  the  SENSO’s  domain.  This  expert  model  is  in 
turn,  used  as  a  comparator  to  determine  each  user’s  skill  level  with  respect  to  that  expert  model. 
The  information  gleaned  from  this  comparison  can  then  be  used  to  optimize  the  interface  to  the 
capabilities  of  the  current  user.  This  report  details  the  processes  and  architecture  proposed  to 

accomplish  this  task. 

The  Integrated  Intelligent  Tutoring  Environment  (IITE)  proposed  by  this  report  has  benefits  m 
that  it  capitalizes  on  existing  tools.  Specifically,  the  more  complex  processes  involved  in  the  de¬ 
velopment  of  this  adaptive  OMI  can  be  accomplished  using  existing  software.  Therefore,  the 
structure  and  design  of  the  adaptive  interface  can  be  the  focus  of  development.  Namely,  we  rec¬ 
ommend  conducting  a  CTA  using  the  Cognitive  Analysis  Tool  (CAT).  CAT  is  a  software  prod¬ 
uct  that  essentially  automates  the  CTA  process.  As  the  CTA  can  be  the  most  time-consuming 
aspect  of  such  an  endeavor,  this  tool  can  result  in  significant  savings.  The  other  difficult  aspect 
'  of  developing  an  adaptive  OMI  is  assessing  the  knowledge  state  of  the  current  User.  This  com¬ 
ponent  can  be  accomplished  using  DSR’s  Integrated  Training  System  (ITS).  Given  the  appropn- 
ate  testing  mechanism,  this  software  can  assess  a  user’s  skill  level  as  compared  to  that  of  an  ex- 
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Edmonds  (1987)  described  five  areas  a  system  can  take  into  account  to  achieve  adaptivity: 

•  user  errors, 

•  user  characteristics 

•  user  performance, 

•  user  goals,  and 

•  the  information  environment. 

The  system  proposed  in  this  report  has  the  potential  to  adapt  based  on  each  of  these  system  areas, 
thus  oroviding  a  great  deal  of  flexibility.  The  expert  model,  which  is  resident  in  the  backpound, 
monitors  the^stem  state  and  the  user's  actions,  determines  the  user’s  intentions  “P°” 

these,  and  determines  if  the  user  needs  assistance.  The  system  also  understands  the  user 
level  End  can  tnilor  that  assistance  accordingly. 

Figure  2-1  is  a  simplified  diagram  depicting  how  this  system  could  function.  The  expert  model 
wiU  be  a  by-product  of  the  CTA.  The  expert  model  can  be  used  to  develop  a  skills  analysis  te 
that  assesses  the  user’s  capabilities  with  respect  to  that  expert  model.  Representations  of  the  ex¬ 
pert  model  and  the  user  model  will  then  be  accessible  to  the  tactical  system. 


Figure  2-1  Adaptive  OMI  Functionality  Diagram 
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3.  PHASE  I  CONCLUSIONS  AND  PHASE  H  RECOMMENDATIONS 

The  review  and  anaivsis  lead  us  to  conclude  that  the  mental  model  obtained  ftom  a  cogmOve  t^k 
analysis  could  provide  a  viable  mechanism  for  adapting  the  operator-machtne  interface  for  the 
M  60R  helicomer  sensor  operator  stauon.  With  the  advances  that  have  been  made  m  cogmuve 
Sle  and  he  advent  of  automated  tools  to  assist  in  the  CTA  process  the  most  complex  ^d 
costrcontponent  of  a  CTA  becomes  more  feasible  at  a  time  when  technology  offers  the  greatest 
potential  for  developing  such  a  capability. 

In  keeping  with  our  recommendation  to  implement  an  adaptive  interface  aboard  *e  SH-MR 
helper  we  propose  that  the  Phase  II  effort  consist  of  prototype  development  ^d  ev^tatton 
of  an  Laptive  OMI.  Specifically,  we  recommend  that  a  prototy  pe  adapttve  interface  be  deve 
Id  to  rspecific  sensor  subsystem.  Since  the  SH-60R  has  not  been  fielded,  we  mcommend 
dLloping  tWs  proof-of-concepton  an  existing  version  of  the  selected  set^or  subsystem  and  to 
be  implemented  in  either  an  embedded  system  on  the  SB-60B  or  a  training  system 

setting. 

We  propose  using  the  Cognitive  Analysis  Tool  (CAT)  to  conduct  the  CTA  on  the  selected  se^or 

subsystem.  This  prototype  will  use  the  output  of  a  cognitive  task  ^ 

required  to  utilize  the  sensor  system.  With  the  assistance  of  an  expert  m  CTA,  a  Subject  Matter 
Expert  (SME)  will  be  trained  to  operate  the  CAT.  The  SME  will  interact  with  CAT  to  specify 
the'^goals,  subgoals,  procedures,  etc.  that  are  performed  on  the  sensor  subsystem,  ^tas 
model  can  then  be  validated  by  other  SMEs  to  ensure  its  accuracy.  Upon  completton,  CAT 
generate  the  task-goal  hierarchy  for  that  subsystem  and  the  underlying  production  system  arc  i- 

lecture. 

The  content  specified  by  this  cognitive  model  generated  by  CAT  will  be  used  to 
narios  that  assess  the  user’s  skill  level.  It  is  proposed  that  the  diagnostic  component  of 
Integrated  Training  System  (ITS)  be  used  to  develop  and  implement  the  sbll  assessment  portion 
of  the  adaptive  interface.  We  specifically  recommend  using  a  current  COTS  authoring  svstem  o 
develop  xL  tests  and  scenarios  that  can  be  associated  with  each  node  of  the  expert  s  task-goal 

hierarchy. 

Once  the  interface  between  CAT  and  the  diagnostic  component  of  the  ITSIhas  bren  created,  the 
heuristics  for  adapting  the  system  can  be  established.  These  heuristics  must  take  into  account 

several  variables  including: 


•  the  current  user’s  skill  level j 

•  the  current  goal, 

•  environmental  circumstances,  etc. 
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It  is  imnerative  that  any  proposed  modifications  to  the  OMI  be  tested  to  ensure  they  achieve  fte 
desired  resuits  To  this  end,  we  propose  to  develop  a  research  plan  that  allowsfor  the  ^aeimUc 
Valuation  of  any  proposed  adaptations.  The  heuristics  for  adapting  the  mterface  wtU  be  bas^ 
on  research  conducted  that  identifies  the  differences  m  knowledge  representafion,  proble 
solving  strategies,  etc.  between  individuals  who  are  considered  experts,  intermediates,  and  nov¬ 
ices  While  there  are  several  theories  on  the  nature  of  these  differences  (e.g^,  Zach^,  1  ), 

this  information  has  not  been  applied  to  an  adaptive  interface  environment  The  ^aptation  heu¬ 
ristics  will  also  be  based  on  an  analysis  of  the  subsystem  to  determine  where  performance  de- 
rements  are  most  likely  to  occur  and,  therefore,  where  the  adaptive  OMI  would  be  most  val  ^ 
To  test  the  various  adaptation  configurations,  we  propose  developing  test  screens  for  som 
prompts  advisories,  etc.  and  testing  their  effectiveness  in  an  operational  or  simulated  situation. 
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4.  ADAPTIVE  OPERATOR-MACHINE  INTERFACE 

This  section  of  the  report  describes  the  foundation  that  underlies  the  proposed  approach  to  match 
the  operator’s  skill  at  various  tasks  and  subtasks  to  the  appropriate  OMI.  This  process  will 
maximize  the  overall  operator-system  performance  through  improved  operator-system  mterac- 
tion.  The  underlying  concept  is  to  identify  a  SENSO’s  skill  level  on  a  particulm  system  or  func¬ 
tion  and,  from  that,  adaptively  change  that  system  or  function’s  OMI  to  match  that  sbll  level. 

There  are  three  components  to  an  adaptive  interface.  These  are. 

•  an  expert  model  that  represents  the  domain  knowledge, 

•  a  user  model  that  represents  the  knowledge  of  the  current  user,  and 

•  a  reasoning  module  that  interprets  user  actions  in  an  attempt  to  infer  their  intentions  and 

modifies  the  interface  features  accordingly. 

These  components  are  encompassed  in  an  architecture  knovra  as  the  Integrated  Intelligent  Tu¬ 
toring  Environment  (IITE).  The  IITE  provides  the  following  benefits  during  development: 

•  an  automated,  state-of-the-art  knowledge  engineering  process  that  results  in  a  sophisti¬ 
cated  expert  model, 

•  advanced  testing  techniques  that  develop  a  roadmap  depicting  the  student’ s  current  un¬ 
derstanding  of  the  domain,  _ 

•  a  rule-based  architecture  that  monitors  the  procedures  conducted  by  the  operator  durmg  a 

watch  and  provides  relevant  and  timely  performance  metrics, 

.  sophisticated,  user-friendly  COTS  authoring  software  to  develop  the  potential  adapta- 

tions,  ,  r  r-  I 

•  a  “middleware”  architecture  that  allows  for  system  reuse  on  multiple  platforms,  tor  mul¬ 
tiple  applications. 

Figure  4-1  depicts  the  high-level  structure  of  the  IITE.  A  CTA  (1)  using  an  automated  tool 
will  result  in  an  expert  model  (2)  of  the  domain.  Ideally,  that  expert  model  will  be  accessible 
to  the  authoring  environment  (3)  such  that  adaptation  frames  can  be  associated  or  linked  to 
the  nodes  of  the  expert  model.  These  structured  frames  (4)  that  contain  the  expert  model,  ad¬ 
aptation  frames,  tests,  and  the  student  model  are  then  presented  to  the  student. 
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Figure  4-1  High-level  arehiteclure  of  the  Integrated  Intelligent  Tutoring  Environment 


The  individual  components  of  the  IITE  are  discussed  in  detail  below. 


4.1  Development  of  an  Expert  Model 

Cognitive  task  analysis  (CTA)  is  a  methodology  for  developing  a  detailed  descripti^  of  the 
knowledge  base  and  cognitive  processing  embodied  in  expert  perform^ce  of  a  task.  ® 

traditional  methods  for  analyzing  a  task  decompose  it  into  subtasks,  skills,  and  knowkdge,  C 
includes  an  analysis  of  the  thought  processes  of  the  task  performer.  According  to  Gordon  and 
Gill  (1997),  a  CTA  provides  specific  insight  into  the  following  types  of  information  as  related  to 


a  person’s  job: 

•  Concepts  and  principles,  and  their  relationship  to  each  other  and  the  task(s). 

•  Task  goals  and  goal  structures,  including  methods  of  achieving  the  goals,  and  imtiat- 
ing  conditions  or  “triggers”  for  goals  and  methods. 


.  Cognitive  skills,  rules,  strategies,  and  plans  applicable  to  the  job. 

•  Perceptual  learning,  pattern  recognition,  and  implicit  or  tacit  knowledge  associated 
with  the  job. 

‘  Figure  4-2  depicts  a  simplified  tree  structure  for  the  job  “Tracking  Targets”  on  a  tactical  com¬ 
puter  console.  The  main  goal  has  several  subgoals  including  “Acquiring  Targets,”  M^toining 
Course,  Speed,  and  Position,”  and  “Classifying  Targets.”  These  subgoals  can  be  further  broken 

down  into  lower  level  goals. 
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Figure  4-2  A  Sample  Task-Goal  Hierarchy 


The  children  of  any  node  can  be  grouped  together  in  three  ways,  either  ANDed.  ORed.  or  both. 
Figure  4-3  depicts  these  three  possible  groupings  of  children  of  a  node. 


Figure  4-3  (a)  fa  an  ORed  grouping  of  ehildren;  (b)  an  ANDed  grouping  of  chHdren;  and  (e) 
an  AND/ORed  grouping  of  children. 
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Noti«  that  in  Figure  4-3  (a)  there  are  2  children  of  node  "A",  and.  unlike  Fi^  4-3  (b),  there  is 
no  arc  connecting  the  links  into  node  A.  This  absence  of  an  arc  means  that  the  cluldren  of  n^e 
"A"  are  ORed.  Either  child  of  node  "A"  contains  the  needed  information  upon  which  node  A  ^ 
is  formulated.  The  arc  connecting  the  links  into  node  "B"  indicates  that  the  chUcten  of  node  B 
axe  ANDed.  All  of  the  children  of  node  "B"  are  required  to  formulate  the  information  repre¬ 
sented  in  node  "B".  Lastly,  Figure  4-3  (c)  depicts  a  combination  of  ANDed  and  ORed  children. 
In  this  case,  the  information  of  node  "C"  can  be  formulated  simply  by  one  single  child  beneath  U 
OR  by  combining  the  information  from  the  other  three  children  which  are  jomed  by  the  AND 

arc. 


To  be  of  use  as  an  aid  to  developing  computer-based  software  for  diagnosing  skill  deficits,  and 
for  designing  decision  support  systems  and  intelligent  tutoring  systems,  the  units  of  knowledge 
that  comprise  the  model  must  conform  to  a  “production  system”  architecture  of  cogmtion  first 
introduced  by  Newell  and  Simon  (1972;  see  also  Newell,  1973).  A  production  system  is  char¬ 
acterized  by  a  set  of  production  rules  in  combination  with  working  memory  and  takes  the  form  of 
IF/THEN  (condition/action)  pairs.  The  IF  (condition)  side  of  the  rule  specifies  the  content  o^^ 
working  memory  and  the  THEN  (action)  side  corresponds  to  overt  behaviors  that  are  “executed. 
By  executing  a  set  of  production  rules,  one  can  simulate  how  to  do  something,  including  com¬ 
plex  tasks. 

The  cognitive  model  of  the  knowledge  required  to  perform  the  task  expertly  is  represented  in  the 
AND/OR  graph  as  previously  described.  One  popular  methodology  for  conducting  a  CTA  is  t  e 
GOMS  process.  This  process  involves  specifying  four  types  of  components: 


•  Goals  -  what  needs  to  be  accomplished; 

•  Operators  -  the  steps  which  must  be  taken  to  accomplish  a  goal; 

.  Methods  -  a  collection  of  steps  which  must  be  executed  in  some  specified  order  to  ac¬ 
complish  a  goal;  and 

•  Selection  rules  -  a  set  of  conditions  that  determine  which  methods  to  select  if  more  than 
one  method  can  be  executed  to  achieve  a  specific  goal. 

Figure  4-4  depicts  the  relationship  of  these  components  in  a  “goal-method  hierarchy.”  Results  of 
a  CTA  represent  a  cognitive  or  mental  model  that  represents  the  knowledge  required  to  perform 
a  task  or  set  of  tasks.  The  top-level  node  in  the  tree  represents  the  top-level  goal/objective.  It 
will  encompass  many  lower  level  components.  Branching  down  from  each  node  are  component 
elements  that  need  to  be  mastered  in  order  to  accomplish  the  node  above.  At  the  lowest  level  of 
the  tree  are  the  “primitive  operators”  or  the  actions  that  cannot  be  decomposed  any  further. 
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SUB  GOAL 


SUB  GOAL 


Selection  Rule 


Alternative  Methods  ^  Method  l 


Method  2 


Operators 


Figure  4-4  Au  Example  of  a  GOMS  Hierarehy  (Ab.traeted  f™”  f 

Newell,  1983;  Kieras,  1988;  Kieeras  and  Poison,  1985,  Williams,  1993) 

Fieure  4-5  depicts  a  high-level  tree  structure  for  the  job  of  the  Sensor  Operator  In  this  tree 
*e  £s.  level  goal  is  to  perform  fte  Sensor  Opemtor  job.  The  first-level  subgoals 

include  4e  different  missions  that  the  SENSO  may  '"8^8® 

fare,  Attti-Surface  Warfare.  Over-the-Horizon  Targeting,  md  .i^f^rlck 

goals  can  be  further  broken  down  into  lower  level  subgoals  or  Mennfy  ^Tmck, 

Lok  a  Track)  and  the  operators  required  to  accomplish  these  subgoals  (e.g..  Ball  Tab  the  Con 

tact  and  Press  the  Hook  FAB). 
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Figure  4-5  High  Level  Hierarchy  for  the  SH-60R  Sensor  Operator 

Gott  (1994)  suggested  that  a  CTA  is  most  appropriate  when  a  task  is  complex,  when  it  is  difficult 
to  learn  because  action  goes  on  in  the  head  of  the  performer,  and  when  tasks  are  not  pre¬ 
sequenced  but  instead  are  dynamic  and  ill-structured.  Each  of  these  qualities  is  characteristic  of 

the  SENSO’s  job. 

4.1.1  Cognitive  Analysis  Tool  (CAT) 

Since  the  level  of  information  provided  by  a  CTA  is  incredibly  detailed,  the  knowledge  acquisi¬ 
tion  process  is  cumbersome  and  labor-intensive.  Moreover,  there  are  a  limited  number  of  indi¬ 
viduals  qualified  to  conduct  such  analyses.  For  this  reason,  Williams  (1993)  created  a  Cognitive 
Analysis  Tool  (CAT)  to  assist  in  the  CTA  process.  CAT  is  an  automated  tool  that  operationa  - 
izes  the  GOMS  analysis  process.  It  can  be  used  by  an  SME  to  develop  the  cognitive  models  rep¬ 
resentative  of  that  expert’s  domain.-The  software  essentially  “walks”  the  user  teough  the  CTA 
process  by  providing  prompts  and  menu  options  with  which  the  user  can  specify  the  goals,  sub¬ 
goals,  and  primitive  operations  required  to  accomplish  each  task. 
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Using  a  “Guidance  Mode.”  the  user  is  constrained  to  follow  a  specified  process  for  capturing 
his/her  knowledge.  After  the  definition  of  a  step,  the  system  queries  the  user  to  determine  if: 

•  the  step  described  requires  that  a  decision  be  made, 

•  the  step  described  requires  that  information  be  stored  in  memory  for  later  retrieval,  or 

•  the  step  requires  that  some  information  be  recalled  from  a  memory  store. 

For  each  of  these  circumstances,  the  user  is  prompted  to  provide  the  appropriate  information  at 
the  appropriate  level  of  detail.  For  example,  if  a  step  description  is  indicative  of  a  decision,  the 
svstem  queries  to  determine  if  the  user's  step  description  implies  that  a  decision  must  be  made. 
If  so  the  system  presents  a  Decision  Step  Dialog  box  similar  to  the  one  shown  in  Figure  4-  . 
This ’dialog  box  imposes  the  IF-THEN-ELSE  structure  in  order  to  identify  the  specifics  of  the 

decision  to  be  made  by  the  decision  step. 


Decision  Step  Diaioq 


Figure  4-6  Decision  Step  Dialogue  Window 

After  all  the  methods  or  set  of  methods  for  the  specified  goals  have  been  defined,  the  system  de¬ 
termines  if  more  than  one  method  has  been  defined  for  a  goal.  For  all  goals  that  have  niore  than 
one  method,  the  user  must  specify  the  selection  rules  -  the  condition  or  set  of  conditions  that 
must  be  met  for  each  method  to  be  triggered. 

In  addition  to  the  goal  hierarchy,  CAT  also  generates  the  production  system  associated  with  each 
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goal  Mora  specifically,  CAT  generates  the  IF-THEN-ELSE  rules  assocmed  wi*  the  spectfied 
Lis  Depentog  on  the  constraints  imposed  by  the  tactical  softwme,  the  production  system 
Lcture  may  be  adaptable  to  a  form  that  can  be  utilized  by  a  tactical  system,  potenbally  reduc- 

ing  implementation  time. 

The  end  result  of  the  CTA  will  be  a  complete  set  of  production  rules  that  represent  Ae  SENSO’s 
mental  model  of  the  goals,  the  tasks,  the  environment,  etc.  This  “expert  model  will  be  used  ^ 
the  ideal  against  which  each  individual  operator  will  be  compared.  This  n^del  could  also  be 
used  to  determine  the  actions  the  user  should  be  taking  at  any  given  time.  That  is,  many  of  the 
“trigoers”  that  activate  a  goal  are  contacts  detected  by  the  system.  If  such  a  tngger  occurs  and 
the  u*ser  does  not  take  any  action,  the  system  could  prompt  the  user  to  begin  the  necessary  proce¬ 
dures.  Therefore,  the  system  could  detect  errors  of  omission  in  addition  to  errors  of  commission. 


4.2  Development  of  a  User  Model 

The  CTA  involving  SMEs  described  previously  will  provide  a  model  of  expert  performance  as 
well  as  provide  task  and  performance  information  for  developing  a  diagnostic  assessment  in¬ 
strument  for  determining  the  skill  strengths  and  weaknesses  of  Sensor  Operators.  As  part  of  Ae 
IITE  development  environment,  a  COTS  authoring  system  will  be  interfaced  to  the  expert  mode 
to  allow  for  the  development  of  test  questions  and  scenarios.  These  diagnostic  instrument  will 
be  used  as  the  basis  for  the  “User  Model”  for  each  user.  The  results  of  these  diagnostic  instru¬ 
ments  will  be  a  knowledge  and  skill  profile  or  “User  Model”  for  each  user.  The  Adaptive  OMI 
Processor  (AOP)  will  modify  the  OMI  with  the  User  Model  to  match  the  capabilities  of  the  par¬ 
ticular  user.  The  skill  level  analysis  technique  that  will  be  used  to  develop  this  User  Model  w^ 
developed  as  a  part  of  DSR’s  Integrated  Training  System  ^TS)  developed  under  SBIRN91-052. 
The  ITS  is  a  training  system  employing  advanced  cognitive  recognition  algorithms  an  em  e  - 
ded  techniques  that  diagnose  trainee  weaknesses  as  part  of  its  adaptive  training  procedure.  It  is 
used  to  train  operators  the  complex  knowledge  and  skills  required  to  effectively  utilize  their  con¬ 
soles.  The  ITS  can  be  used  to  maps  the  skills  of  the  operator  (the  user  model)  to  the  expert 

model. 

The  assessment  mechanism  must  be  detailed  enough  to  develop  a  comprehensive  User  Model. 
The  assessment  would  probably  consist  of  a  series  of  scenarios  that  progress  in  complexity,  each 
focusing  on  one  or  more  of  the  goals  contained  in  the  expert  model.  The  two  critical  elements 
required  for  an  effective  skill  analysis  are:  a)  identifying  appropriate  situational  factors  that  in¬ 
fluence  SENSO  performance  (Ikomi  &  Tirre,  1994;  Jacobs  &  Dempsey,  1993),  and  b) 
ing  appropriate  performance/process  measures  and  criteria  (Konrad,  Kramer,  &  Watson,  1994; 
Dickinson  &  Teachout,  1993;  Frenkel,  1996).  By  incorporating  key  situational  factors  mto  sce¬ 
narios  or  simulations  used  to  assess  the  skills  of  SENSOs,  the  predictive  ability  of  the  overall  as¬ 
sessment  instrument  is  enhanced.  For  example,  providing  scenarios  involving  multiple  mcoming 
enemy  targets  that  must  be  identified  and  prioritized  may  result  in  task  overload  for  some  Sensor 
Operators,  thus  reducing  their  performance.  Alternatively,  partial  commumcation  failures  (due 
to  equipment  malfunction)  or  communications  that  are  incongruent  with  sensor  readings  may 
cause  momentary  confusion  and  performance  breakdowns.  Skill  deficits  can  be  determined  in  a 
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reliable  manner  by  comparing  performance  deficits  across  scenarios  and  by  looking  at  p^ess 
variability  of  individuals  relative  to  the  CTA-generated  expert  model,  mre  are  a  number  of 
performance  indicators  that  can  be  measured  and  assessed  that  will  provide  the  necess^infor- 
mation  to  accurately  diagnose  skill  deficiencies.  For  example,  time  lag  and  n^ber  and  type  of 
errors  between  initial  stimulus  event  and  response  may  provide  valuable  performance  informa¬ 
tion 


The  activities  required  for  developing  a  SENSO  skill  diagnostic  instrument  include: 

.  Conduct  interviews,  surveys,  etc.  with  SMEs  as  well  as  non-SMEs  for  the  purpose  of 
identifying  key  situational  factors  that  influence  performance  and  as  a  pre-cursor  to  de¬ 
veloping  scenarios/simulations  that  can  be  integrated  into  the  skill  assessment, 

•  Develop  and  validate  the  skill  assessment  scenarios/simulations  as  well  as  the  associated 

performance  and  process  measures, 

.  Implement  the  scenarios  into  the  ITS  so  that  the  User  Model  can  be  created. 


Ideally,  the  authoring  system  component  of  the  IITE  can  also  be  used  to  present  the  scenmos. 
This  system  has  templates  for  creating  multiple-choice  questions  as  well  as  performance-based 
scenarios.  These  assessment  instruments  are  linked  to  the  nodes  in  the  expert’s  task-goal  hierar¬ 
chy  such  that  the  User  Model  will  reflect  the  user’s  mental  model  as  compared  to  that  of  the  ex¬ 
pert.  Figure  4-7  depicts  this  relationship  between  the  expert  and  the  User  Model.  Note  that  the 
User  Model  is  created  as  a  subset  of  the  expert  model.  Figure  4-7  shows  a  hypothetical  expert 
model  and  a  User  Model  of  the  same  domain.  Note  that  the  user’s  model  consists  of  the  nodes  of 
the  expert  model  but  the  skill  assessment  test  determined  which  of  those  nodes  the  user  under¬ 
stands. 


Figure  4-7  Depiction  of  the  Comparison  Between  an  Expert  and  User  Model 
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4.3  Development  of  the  Adaptive  Operator  Machine  Interface  Processor 
The  Adaptive  OMI  Processor  (AOP)  will  utilize  the  User  Model  and  a  set  of  Adaptive  OMI  rules 
to  form  an  initial  Adaptive  OMI  configuration.  This  AOP  will  identify  what  changes  should  be 
made  to  the  system’s  OMI  in  order  for  the  current  SENSO  to  more  effectively  perform  the  re¬ 
quired  tasking. 

The  task  of  establishing  rules  for  improving  specific  weaknesses  in  an  operator’s  knowledge 
should  be  based  on  studies  showing  improvements  in  overall  performance  as  a  function  of  spe¬ 
cific  OMI  changes.  These  features  should  be  proven  to  enhance  performance  in  designated  areas 
of  weakness.  An  expert  operator  would  be  expected  to  outperform  the  inexperienced  operator 
that  is  given  these  aids,  but  there  is  strong  evidence  that  providing  this  machine  level 
"mentoring”  enhances  the  level  of  performance  from  inexperienced  operators.  The  results  of  this 
effort  will  quantify  the  improvements  achievable. 

The  AOP  will  be  designed  to  adapt  the  OMI  based  upon  the  results  of  the  skill  level  analysis  and 
a  predefined  set  of  context-sensitive  aids  established  and  tested  to  ensure  that  the  desired  en¬ 
hanced  operator  effectiveness  and  performance  are  achieved.  For  example,  if  the  operator  ex¬ 
hibited  difficulty  in  performing  a  task  -  like  updating  track  histories  -  this  would  show  up  as  a 
weak  method  in  the  User  Model.  The  associated  context-sensitive  advisories  would  be  mapped 
by  the  processor  for  the  OMI  to  adapt  by  providing  a  checklist  of  correct  steps  and  concepts  re¬ 
lated  to  the  knowledge  required  to  perform  this  task.  In  most  cases,  the  OMI  would  be  adapted 
to  provide  additional  advisories,  prompts,  or  aids  that  will  help  the  operator  do  a  better  job  in  the 
areas  in  which  experience  is  lacking.  The  capability  would  be  available  to  limit  specific  actions 
that  could  result  in  serious  impacts  to  mission  success  or  damage  to  equipment. 

The  SH-60R  has  been  designed  to  provide  the  operator  with  a  great  deal  of  functionality  that  will 
assist  in  accomplishing  the  requisite  tasks.  Features  such  as  the  Weapon  and  Sensor  Envelope 
display  will  allow  the  novice  operator  to  make  effective  tactical  decisions.  Additionally,  there  is 
a  proposed  Decision  Support  System  (DSS)  that  will  provide  even  more  tools  to  aid  the 
SENSO’s  decision-making  process.  A  user  model  that  describes  the  skill  level  of  the  user  will 
also  provide  valuable  information  about  that  user’s  decision-making  capability,  and  therefore, 
his/her  reliance  upon  the  tools  resident  within  the  system.  This  User  Model  information  can  be 
used  to  provide  an  automatic  invocation  of  those  tools  for  the  novice  user.  Specifically ,  it  is  en¬ 
visioned  that  if  the  DSS  comes  to  fhiition,  the  AOP  could  be  interfaced  to  it  such  that  it  will  in¬ 
voke  the  required  DSS  feature  when  the  User  Model  indicates  it  is  necessary.  Figure  4-8  shows 
the  possible  relationship  between  the  AOP  and  the  DSS.  Once  the  expert  model  is  developed, 
goals,  subgoals,  or  procedures  that  are  aided  by  some  DSS  functionality  can  be  identified  (the 
goal  structures  with  bold  outlining  in  Figure  4-8.  If  a  user  is  weak  in  that  particular  goal,  the 
DSS  could  be  automatically  activated. 
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Embedded  coaching  is  on-line  assistance  that  could  be  used  when  the  system  infers  that  the  user 
has  performed  a  procedure  incorrectly,  missed  an  important  event,  or  is  performing  an  unneces¬ 
sary  procedure.  This  would  include  the  pre-programmed  cues,  prompts,  advisories,  and  help 
screens  that  will  assist  the  user  in  overcoming  any  performance  obstacles. 

Several  representative  system  help  aids  must  be  defined  that  provide  operator  cues  in  order  to 
improve  her/his  ability  to  perform  the  mission  effectively.  Figure  4-9  shows  several  display  help 
items  that  were  designed  for  adapting  the  OMI.  These  examples  show  a  simplistic  system  pres¬ 
entation  feature  that  is  automatically  adapted  as  the  operator  performs  the  functions  associated 
with  these  presentation  items.  More  complex  features  and  graphics  can  be  associated  with  these 
items  and  presented  to  the  operator  as  audio  and/or  graphics  overlays. 
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An  adaptive  OMI  could  also  prove  valuable  by  providing  access  to  electronic  resources.  The 
sensor  operator  uses  many  sources  of  information  to  successfully  complete  the  various  tmsaons 
and  must  carry  in  the  aircraft  the  many  manuals/reference  materials  that  may  be  required.  Ty^- 
cally,  these  references  are  selected  prior  to  flight,  after  the  mission  prebrief.  The  adaptive  OMI 
system  could  be  designed  to  unburden  the  operator  and  make  him  better  able  meet  the  umque 
demands  of  the  operating  environment  by  providing  immediate  access  to  the  required  informa¬ 
tion  electronically.  However,  this  information  in  its  “raw”  form  could  be  overwhelmmg  to  a 
novice.  Therefore,  the  OMI  could  also  individualize  the  presentation  of  that  information  to  the 
knowledge  level  of  the  current  user.  For  example,  the  experienced  user  could  be  presented  with 
information  summaries  (e.g.,  graphics  or  tables),  while  the  novice  may  require  the  prerequisite 
information  that  went  into  building  the  graphic  or  table.  Some  information  that  could  be  acces¬ 
sible  to  the  SENSO  includes  the  Electronic  Order  of  Battle,  the  Rules  of  Engagement,  etc. 

Table  4-1  delineates  the  potential  ways  in  which  users  could  be  provided  with  assistance  de¬ 
pending  upon  their  individual  categorization  as  an  expert  or  a  novice.  This  table  is  not  all- 
inclusive  but,  instead,  presents  a  few  representative  samples.  Basically,  the  expert  has  a  compre¬ 
hensive  representation  of  the  domain  while  the  novice  may  only  understand  basic  facts  or  con¬ 
cepts.  Presentation  of  information  to  help  the  novice  should  attempt  to  strengthen  the  links  m  the 

novice’s  representation. 


Adaptive  OMI 
Assistance 

Expert 

Knowledge  representation  con¬ 
sists  of  highly-elaborated  con¬ 
ceptual  structure  or  mental  model 
based  on  underlying  generative 
principles 

Novice 

Knowledge  representation  consists 
of  basic  facts,  concepts,  in  de¬ 
clarative  form 

Help 

Clear  succinct;  upon  request  only 

Expanded  help;  upon  request  or 
when  needed 

Checklists 

Statement  of  steps 

Statement  of  steps  with  cues  to 
appropriate  button  actions 

Prompts 

If  time  lag  for  e.xecution  occurs 

When  User  Model  indicates  weak 
skill;  if  time  lag  for  execution  oc- 
curs  _  ■ 

Presentation 
of  Electronic 
Information 

Upon  request  only 

When  User  Model  indicates  weak 
skill;  upon  request 

Table  4-1  Sample  Adaptation  Rules 
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5.  SYSTEM  ARCHITECTURE 

5.1  Conceptual  Architecture 

The  proposed  system  architecture  is  founded  on  the  three-tier  model.  This  model  allows  for  the 
isolation  of  the  User  Interface  from  the  adaptation  components  and  any  storage  mechanisms 
used.  This  type  of  approach  also  allows  for  a  tremendous  amount  of  software  re-use  by  isolating 
key  components  of  the  system.  An  example  of  this  of  architecture  is  found  in  Figure  5-1. 

The  three-tier  model  is  broken  into  the  following  components: 

5.1.1  User  Interface  Tier 

•  Multiple  User  Interfaces  Required 

-  CAT 

-  Authoring  Tools 

-  Tutoring 

-  Training 

-  System 

•  User  Interfaces  Should  Leverage  Off  Of  T raining  Tier 

Provides  Maximum  De-Coupling  Of  User  Interfaces  With  Training  Logic 

5.1.2  Training  Tier 

•  All  Logic  Resides  In  This  Tier 

-  Adaptive  Functions 

-  Authoring  Functions 

-  Training  and  Assessing  Functions 

•  Middleware  Component  Provides  Stable  API  For  Adaptive  Training  Components 

-  Provides  For  The  Isolation  Of  Adaptive  Logic  From  The  Base  Hardware  and  Operating 
System(s) 

-  Allows  Development  Independent  Of  System  Hardware 

-  Provides  A  Structured  Mechanism  For  The  Support  Of  ‘Technology  Refreshment’ 

-  Provides  Growth  Path  For  Both  New  and  Legacy  Systems 

•  Logic  Is  Isolated  From  Other  System  Components 

•  Provides  Structure  For  Maximum  Re-Use  Of  Components 

5.1.3  Database  Tier 

•  Repository  For  All  Data 

•  Isolation  Allows  For  A  Hardware  Independent  Architecture 

•  Makes  Data  Available  For  All  Training  Tier  Components 

•  Provides  Flexibility  For  Future  Storage  Mediums 

-  Smart  Cards 

-  PC  Cards 

-  IR/Wireless  Technologies 
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The  proposed  system  architecture  involves  the  follovving  aspects: 

•  Population  of  knowledge  database  with  an  expert  model. 

•  Analysis  of  a  trainee/student  in  relation  to  the  knowledge  database. 

•  Inference  Engine 

•  Training  Middleware 

Using  a  tool  like  the  Cognitive  Analysis  Tool  (CAT),  the  subject  matter  expert  can  develop  an 
expert  model  that  populates  a  given  knowledge  database  with  specifics  about  a  given  task.  A 
trainee/student  would  then  undergo  an  evaluation  in  relation  to  the  expert  s  knowledge  base.  To 
successfully  perform  this  task  we  recommend  that  a  series  of  questions/scenarios  be  presented  to 
the  user  such  that  a  determination  can  be  made  about  his/her  knowledge  of  a  given  subject  area. 
To  this  end,  a  tool  like  DSR’s  Integrated  Training  System  (ITS)  could  be  used.  ITS  provides  an 
efficient  method  for  assessing  a  student’s  current  knowledge  of  a  given  task.  The  output  of  this 
process  would  be  a  User  Profile  which  contains  all  the  details  necessary  for  correlating  a  User’s 
skill  level  as  compared  to  that  of  an  Expert. 

The  above  tasks  provide  a  means  to  evaluate  a  given  user  against  an  expert  and  determine  how 
the  interface  should  be  adapted  to  meet  his/her  specific  needs.  In  order  to  complete  the  Adaptive 
OMI  process,  a  software  inference  engine  will  run  on  the  tactical  system  itself  This  engine  will 
make  inferences  based  upon  scenario  triggers  as  well  as  operator  actions  by  making  an  evalua¬ 
tion  of  the  student  User  Profile  versus  the  expert  knowledge  base.  The  output  of  this  process 
would  be  the  adaptation  of  the  interface  to  meet  the  users  specific  needs. 

The  final  component  of  the  proposed  architecture  is  the  Training  Middleware  layer.  This  layer  is 
meant  to  separate  the  adaptive  and  training  components  from  both  the  hardware  specifics  and  the 
underlying  user  interface  application.  By  providing  a  middleware  component,  the  entire  archi¬ 
tecture  becomes  portable  across  hardware  and  system  platforms.  In  addition,  it  provides  a  stable 
application  programmer’s  interface  (API)  for  building  future  user  interfaces.  By  writing  these 
future  user  interfaces  to  the  Training  middleware  API,  systems  will  be  able  to  take  advantage  of 
this  type  of  adaptive  user  interface  and  training  capabilities  without  having  to  re-develop  them  on 
their  own. 

Use  of  Training  Middleware  provides  for  the  following: 

•  Provides  For  The  Isolation  Of  Adaptive  Logic  From  The  Base  Hardware  and  Operating  Sys- 
tem(s) 

•  Allows  Development  Independent  Of  System  Hardware 

•  Provides  A  Structured  Mechanism  For  The  Support  Of  ‘Technology  Refreshment’ 

•  Provides  Growth  Path  For  Both  New  and  Legacy  Systems 
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In  previous  systems  developed  by  DSR,  such  as  the  Multi  Purpose  Proce^r  (WP)  syste^  use 
of  Middleware  has  proved  that  the  above  principles  can  be  realized.  To  date,  tot  system  has 
been  potted  to  five  different  hardware  platforms  and  four  different  Operatmg  Systems.  In  al 
cases,  the  amount  of  code  tot  Application  Developers  were  required  t^hauge  ™  'f' ' 

the  amount  of  code  change  to  the  Middleware  layer  was  less  than  1%.  This  h^  flowed  tot  pro¬ 
gram  to  truly  isolate  the  work  of  Application  Developers  from  the  decision  of  which  hardw  are 

and  operating  system  to  support. 


Figure  5-1  System  Architecture 
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6.  IMPLEMENTATION  AND  TRAINING 


6.1  IMPLEMENTATION 

The  true  value  of  an  adaptive  OMI  will  be  realized  only  if  is  used  throughout  the  SENSO’s  ca- 
reer  To  provide  maximum  exposure  of  the  OMI  to  the  user  and,  in  the  corollary,  the  user  to  the 
OMI,  the  OMI  should  be  introduced  in  the  first  phase  of  SENSO  traimng  as  a  specific  cumeu- 

lum  item. 

Following  initial  accession  training,  the  OMI  should  form  the  core  capability  in  the  Fleet  Re¬ 
placement  Squadron  (FRS).  FRS  training  should  provide  the  new  SENSO  ^ith  maxim^  exp 
sure  to  every  reasonably  possible  tactical  environment  and  scenario  and  build  his/her  skills  data¬ 
base  as  much  as  possible  before  assignment  to  an  operational  squadron.  To  provide  eiAanced 
training  for  the  SENSO  “community”,  the  FRS  should  carefully  examine  the  feasM- 
ity/desLbility  of  using  the  OMI  capability  to  monitor  FRS  training  itself  In  addition  the  FRS 
should  address  issues  surrounding  OMI  control,  such  as  maintaimng  master  skill  level  and  indi- 

vidual  student  profile  databases. 

The  OMI  should  be  used  in  all  squadron  operations.  Whether  during  predeployment  training  or 
embarked  training/operations,  the  OMI  system  embedded  in  the  SENSO’s  workstation  will  be 
the  only  continuously  available  and  non-subjective  capability  to  measure  operator  performance. 
OMI  should  be  included  as  a  critical  functionality  for  all  related  task  trainers,  whether  as  part  of 
a  training  capability  embedded  in  the  aircraft  or  in  stand-alone  systems. 

The  OMI  skill-level  database  must  maintain  its  identity  with  the  respective  individual  SENSO 
operator  throughout  his/her  career.  To  that  end,  the  database  must  transfer  with  the  individual 
with  all  the  import  as  that  person’ s  service  record. 


6,2  System  Training 

Although  the  goal  of  an  adaptive  OMI  is  to  simplify  a  system  for  the  user,  it  will  still  require  ex¬ 
tensive  training  to  ensure  the  user  understands  the  adaptive  functions.  The  user  must  be  trained 
to  operate  an  interface  with  the  potential  to  change  constantly.  Krogsaeter  Oppermaim,  and 
Thomas  (1994)  recommended  that  training  should  include  an  introduchon  to  the  rationale  behind 
the  adaptation.  In  this  way,  instead  of  training  on  the  potential  adaptive  screens  the  user  is  pro- 
vided  with  an  understanding  of  the  underlying  logic. 

Adaptive  OMI  system  training  needs  to  be  embedded  in  every  phase  of  sensor  operator  naming, 
from  elementary  familiarity  at  the  accession  level,  to  full  system  understanding  m  the  FRS  and 
operational  squadron.  Training  for  OMI  system  maintenance  needs  to  be  incorporated  m  the  re¬ 
spective  equipment  training  conducted  in  the  FRS  for  both  organizational  and  intermediate  level 

maintenance. 

The  OMI  systems  must  also  be  incorporated  in  the  respective  SH-60R  NATOPS  and  its  use  and 
functionality  be  made  an  integral  part  of  squadron  training. 
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6.3  Configuration  Management 

DSR’s  CM  Program  utilizes  an  integrated  product  team  approach  to  creating,  managing  and 
controlling  all  products  developed  under  our  contracts.  DSR’s  CM  Program  requires  overall 
project  p^icipation  by  all  the  engineering  and  service  organizations  All  the  project  p^icip^ts 
(Program  Management,  SW  Engineering,  Systems  Engmeenng,  Quality  Assmance,  Integrated 
Logistics  Support,  Testing  &  Integration)  comprise  an  informed  and  involved  umfied  project 
team.  The  CM  Program  at  DSR  is  always  coordinated  and  consolidated  with  the  corresponding 
CM  Program  of  our  subcontractor,  if  applicable. 

The  CM  Program  includes  controlled,  tiered,  on-line  libraries  for  SW  development,  an  on-line 
Trouble  Reporting  System  for  cataloging,  correcting  and  reporting  defeats  ^  on-line  datable 

for  CM  data  collection,  mechanisms  for  identification  and  control  of  all  HW  an 

Configuration  Control  Board  (CCB)  for  both  HW  and  SW,  and  a  documented  product  rde^e 
and  delivery  process.  The  specifics  of  the  CM  Program  can  be  found  in  the  appropnate  CM  Plan 
and  its  derivative  documentation. 


6.4  Data  Management 

The  data  management  effort  is  a  subset  of  the  CM  Program.  It  is  conducted  in  accord^ce  with 
the  documented  Data  Management  Procedures.  All  data  produced  on  the  project  will  be  umquely 
identified,  managed  and  controlled,  and  appropriately  baselined  for  archivmg  and  delivery  i^ing 
the  on-line  CM  System  mechanisms  as  appropriate.  All  data  received,  created  and  delivered  on 
the  project  is  managed  within  this  established  DM  process. 

6.5  Software  Quality  Assurance 

A  Quality  Assurance  (QA)  program  is  conducted  on  the  project  by  a  designated  QA  represenm- 
tive.  The  QA  function  operates  on  behalf  of  the  corporation.  Each  software  project  is  required  to 
conduct  its  activities  in  accordance  with  a  documented  and  approved  engineering  process.  QA 
personnel  ensure  that  the  project  follows  its  documented  process  and  that  artifacts  exist  on  the 
project  to  demonstrate  compliance  for  purposes  of  accountability  and  audit.  QA  has  foil  and 
open  access  to  all  project  data  for  purposes  of  reviewing  and  auditing  its  activities.  The  Program 
Manager  is  responsible  for  ensuring  that  QA  has  appropriate  access  to  all  project  data  as  re- 
quired.  QA  conducts  and  documents  periodic  evaluations  and  audits  during  the  life  cycle  of  the 

project. 
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7.  FUTURE  DIRECTIONS 

7.1  Investigate  Potential  to  Apply  to  Multiple  Stations  (AOMI  in  a  Distributed  Environ¬ 
ment) 

A  sophisticated  AOMI  could  also  allow  for  consideration  of  multiple  operators  or  computer  ac¬ 
tions  in  a  distributed  environment.  Each  operator’s  AOMI  could  be  sensitive  to  the  performance 
and  skill  level  of  the  other  operators  within  the  aircraft  and  reflex  accordingly  to  provide  a  com¬ 
plementary  capability.  This  complementary  capability  could  provide  the  ability  for  the  AOMI  to 

operate  in  a  system  environment. 

Additionally,  as  the  SENSO  is  performing  in  a  tactical  data  link  environment  (Link  11/16)  the 
AOMI  could  be  reactive  to  other  platform  actions  or  information  provided  from  these  external 
participants.  The  AOMI  could  automatically  solicit  additional  operator  or  computer  information 
via  the  tactical  Link  if  it  senses  the  SENSO  could  be  aided  by  this  additional  information. 

7.2  Investigate  Potential  for  Real  Time  Updates  to  User  Model 

A  flexible  AOMI  could  allow  for  changes  in  its  functionality  based  on  changes  in  operator  per¬ 
formance  and  skill  level.  This  flexibility  can  be  dynamic  to  readily  respond  to  changes  m  op¬ 
erator  performance  in  real  time.  The  AOMI  could  monitor  the  operator’s  performance  during  a 
particular  mission  and  adjust  its  functionality  as  it  detects  fatigue,  stress,  information  overload, 
or  conversely  as  it  detects  increases  in  the  operator’s  performance  level  due  to  proficiencies 
gained  and  skills  learned  during  a  particular  mission  or  exercise. 

7.3  Potential  Application  to  Common  Platforms 

The  final  architecture  proposed  for  the  AOMI  will  be  modular  and  structured  in  a  middleware 
concept.  As  a  result,  this  architecture  can  be  transportable  to  most  platforms  providing  a  consis¬ 
tent  AOMI  capability  as  we  approach  a  common  cockpit  design  for  Naval  aircraft.  A  consistent 
AOMI  would  minimize  life-cycle  and  training  logistics  across  platforms. 

7.4  Development  of  an  Operator  Workload  Evaluation  System 

An  important  variable  that  should  be  considered  for  the  design  of  any  interface  is  the  amount  of 
workload  experienced  by  the  operator.  Cognitive  process  measures  could  be  used  to  determine 
individual  capability  with  regard  to  overall  processing  capacity,  ability  to  focus  attention  when 
confronted  with  multiple  distracters,  ability  to  share  attentional  resources  across  various  sources, 
and  the  ability  to  quickly  and  efficiently  switch  attention  in  response  to  changing  situation  dy¬ 
namics. 

During  times  of  intense  workload,  even  the  most  experienced  operator  may  display  perfomi- 
ance  decrements.  However,  the  measurement  of  workload,  particularly  in  an  on-line  envi¬ 
ronment,  can  be  a  difficult  task.  Frenkel  (1996)  described  a  Pilot  Evaluation  System  (PES) 
developed  as  a  screening  device  for  potential  pilots.  The  system  is  a  self-contained,  autono¬ 
mous  system  that  performs  evaluations  objectively  -  without  instructor  intervention.  The 
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PES  consists  of  17  3-minute  scenarios  that  progress  in  complexity  from  simple  to  complex. 
As  scenario  complexity  increases,  the  candidate’s  workload  also  increases  typically  causmg 
a  performance  decrement.  The  PES  was  developed  based  partially  on  the  information  ob¬ 


tained  from  a  CTA. 

This  system  is  designed  to  measure  various 


cognitive  process  measures  including: 


•  the  individual’s  capability  with  regard  to  overall  processing  capacity, 

•  the  ability  to  focus  attention  w'hen  confronted  with  multiple  distracters, 

•  the  ability  to  share  attentional  resources  across  various  sources,  and 

•  the  ability  to  quickly  and  efficiently  switch  attention  in  response  to  changing  situation 


dynamics. 

While  the  measurements  obtained  from  this  system  have  been  successfully  applied  to  the  selec¬ 
tion  of  potential  ATC  candidates,  similar  measures  could  also  be  applied  to  the  desi^  of  ^ 
adaptive  OMI.  More  specifically,  a  Workload  Evaluation  System  (WES)  could  be  developed  to 
identify  processes  or  procedures  that  are  consistently  difficult  for  most  users  to  complete  because 
of  the  workload  demands  they  impose.  In  circumstances  such  as  this,  the  interface  could  be 
modified  to  automate  some  component  of  the  process  or  present  additional  formation  to  help 
the  operator  get  through  the  difficult  process  by  reducing  the  required  workload. 


7.5  Intelligent  Tutoring  System 

As  shown  in  Figure  7-1,  the  domain  knowledge  specified  for  the  adaptive  OMI  can  also  be  used 
as  the  knowledge  base  for  Intelligent  Computer-Assisted  Instruction  (ICAI)  or  for  an  Expert 

System. 


Figure  7-1  Other  Applications  of  the  Domain  Knowledge  Base 
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In  ICAI,  the  courseware  would  be  structured  according  to  the  task-goal  hierarchy.  More  specifi¬ 
cally,  each  node  could  contain  information  representing  the  information  content  to  be  mcorpo- 
rated  in  an  exercise.  An  exercise  could  consist  of  a  packet  of  information  that  specifies: 

•  the  goal  or  objective  of  the  exercise, 

.  an  exposition  of  the  information  content  and  the  inference  which  is  made  given  that  in¬ 
formation, 

•  a  problem  which  requires  that  the  student  apply  the  inference  which  is  presented  in  the 
exposition,  and 

.  a  set  of  diagnostics  which  test  the  students'  knowledge  about  the  elements  of  information 
making  up  the  exercise  exposition. 


Figure  7-2  presents  a  sirrtplified  graphic  of  a  partial  AND/OR  graph  with  its  exercises  portrayed 
as  nodes. 


Figure  7-2  The  Composition  of  Exercises  in  a  Partial  Graph 


Exercise  D  in  this  case  might  portray  the  objective  of  performing  an  addition  of  two  counting 
numbers  such  as  2+2=4.  In  order  to  perform  this  simple  task,  one  needs  to  understand  the  con¬ 
cept  of  a  number  that  is  represented,  for  example,  in  Exercise  A.  The  concept  of  sumrnation,  as 
indicated  by  the  plus  operator  "+”,  represented  in  exercise  B;  and  the  concept  of  equality,  as  m- 
dicated  by  the  equal  sign  (i.e.,  =)  as  represented  in  Exercise  C.  Each  of  the  exercises  A,  B,  and  C 
may  be  further  broken  down  into  those  attributes  which  make  up  their  respective  concepts.  The 
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exposition  of  Exercise  D  would  provide  several  examples  of  addition  and  how  the  concepts  in 

Exercises  A  B  and  C  are  put  together  to  formulate  the  equations  portrayed.  Followmg  the  ex¬ 
position  of  the  exercise  objective  and  the  way  in  which  the  concepts  which  were  learned  previ¬ 
ously  are  put  together  to  formulate  a  solution  to  our  equation  of  the  form  2+2-?,  the  student  is 
required  to  solve  a  set  of  problems  of  a  similar  form.  If  the  student  can  successfully  solve  the 
problems  posed,  then  one  can  infer  that  the  student  knows  how  to  compute  summations  of  this 
form.  If  the  student  fails  to  solve  the  problems  posed,  then  the  set  of  diagnostics  are  presented  to 
the  student  to  determine  which  concepts,  as  represented  in  Exercises  A,  B,  or  C,  the  student  is 
having  difficulty  with.  This  simple  example  is  meant  to  describe  the  components  of  an  exercise 
associated  with  each  node  in  the  graph  and  not  meant  to  minimize  the  difficulty  with  which  an 
instructor  or  tutor  is  faced  when  attempting  to  teach  such  abstractions  as  the  concepts  of  number, 

addition  and  equality'. 

A  complete  cognitive  model  consists  of  all  of  declarative  facts  and  the  ways  m  which  they  are 
combined.  These  declaratives  can  be  combined  to  form  concepts  and  condition-action  pairs 
called  rules  which  govern  how  system  components  interact  to  produce  a  ^ction;  condition- 
action  rules  which  prescribe  what  action  to  take,  given  a  certain  set  of  conditions;  or  condition- 
action  rules  which  prescribe  what  operation  to  apply  (i.e.,  as  in  the  addition  example)  given  a 
specific  set  of  symbols.  Virtually  any  domain  of  knowledge  can  be  structured  in  this  way  to 
form  a  model  of  the  knowledge  which  must  be  acquired  in  order  to  perform  a  specific  task.  Each 
node  representing  an  exercise  may  also  be  associated  with  alternative  portrayals  of  that  informa¬ 
tion  employing  multimedia  and/or  may  be  associated  with  alternative  problems.  These  alterna¬ 
tive  problems  could  be  used  to  provide  practice  in  the  application  of  a  fact,  concept,  or  rule, 
which  is  represented  by  the  node.  The  alternative  media  portrayals  of  the  information  repre¬ 
sented  in  a  node  could  also  be  used  to  facilitate  how  well  someone  understands  the  information 

content  of  a  node. 


7.6  Expert  System 

An  expert  system  could  be  developed  using  the  knowledge  base  of  the  expert  SENSO  and  a  set 
of  rules,  which  define  how  this  expert  uses  the  knowledge  base  to  solve  problems.  An  expert 
system  lias  a  great  deal  of  applicability  to  the  SH-60R  sensor  operator.  It  could  provide  indi¬ 
vidualized  assistance  in  tasks  that  require  a  decision.  The  expert  system  could  use  the  estab¬ 
lished  production  system  architecture  to  infer  the  most  appropriate  decision  based  on  the  current 
task  environment. 

7.7  Note-Taking  Utility 

Sensor  operators  use  several  paper-based  resources  in  their  day-to-day  work.  One  such  resource 
includes  a  “gauge  sheet.”  A  gauge  sheet  is  essentially  a  set  of  personalized  notes  that  the 
SENSO  has  found  useful  for  conducting  his/her  job.  If  the  tactical  system  allows,  one  potential 
option  is  to  provide  the  capability  for  the  user  to  view,  and  possibly  record,  personalized  notes 
for  use  in  a  job  task. 
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If  a  note-taking  capability  was  embedded  in  the  application,  the  user  could  immediately  input 
any  information  that  he  or  she  learned  from  a  coaching  pop-up  window.  In  fact,  it  could  be  ex¬ 
tremely  useful  if  the  SENSO  could  “copy”  text  and  graphics  from  a  reference  and  “paste”  them 
into  his  or  her  on-line  gauge  sheet. 

7.8  COTS  Applicability 

COTS  hardware  and/or  software  based  prototype  and  production  systems  are  attractive  hosts  for 
integrated  training  systems  and  Adaptive  OML  COTS  based  simulators,  training  techniques  and 
Adaptive  OMI  interfaces  can  be  efficiently  rehosted  on  production  systems  providing  new  func¬ 
tionality  for  improved  sensor  operator  performance.  The  on-line  embedded  training  systems  can 
be  used  for  initial  training  and/or  for  upgrading  operator  skills  on  a  continual  basis  widi  recorded 
or  simulated  data  inputs.  Measurements  of  skill  levels  with  various  system  configurations  can  be 
used  to  recommend  system  OMI  appropriate  to  the  skill  level  of  the  individual  operator  so  that 
total  system  performance  is  enhanced.  This  capability  would  be  particularly  useful  in  dynamic 
high  contact  rate  littoral  air  ASW  environments  where  operator  skill  levels  at  configuring  buoy 
and  processing  modes  and  decision  processes  through  detection,  classification,  tracking  and  lo¬ 
calization  can  have  critical  impact  on  mission  performance. 
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